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Final  Technical  Report 


The  research  efforts  supported  by  AFOSR  Grant  AFOSR-84-0381  were  directed 
towards  development  and  analysis  of  robust  estimation  techniques  for  autoregressive 
(AR)  and  autoregressive-moving  average  (ARMA)  models.  Work  on  related  system 
theoretic  problems  associated  with  parameter  estimation  problems  for  time  series  models 
and  on  square- root  filtering  for  least  squares  state  estimation  applications  was  also  car¬ 
ried  out.  Finally,  an  adaptive  estimation  technique  for  a  class  of  piecewise  (in  time)  sta¬ 
tionary  signals  was  developed.  The  motivation  for  our  research  arises  from  applications 
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state  estimation  (Kalman  filtering),  and  spectral  analysis.  The  general  goal  of  this 
research  has  been  to  put  together  ideas  and  techniques  from  statistics,  signal  processing, 
and  system  theory  to  bring  new  perspectives  to  such  problems 

Our  research  on  various  autoregressive  modeling  problems  resulted  from  a  desire  to 
relax  some  of  the  assumptions  made  by  previous  researchers,  in  order  to  broaden  the 
domain  of  application  of  the  basic  technique  which  has  proved  to  be  useful  in  a  range  of 
signal  processing  tasks.  In  particular,  our  efforts  have  been  directed  at  the  goal  of 
obtaining  allowing  robust  estimates  in  the  presence  of  outliers  in  the  observed  signal  and 
in  modeling  of  signals  whose  spectral  characteristics  change  abruptly  from  time  to  time 
These  two  situations  provide  models  that  have  wide  applications  to  signal  processing 
problems  encountered  in  practice.  First,  outliers  represent  a  particularly  deleterious  form 
of  noise  with  respect  to  destroying  the  quality  of  least  squares  estimates  (as  is  well 
known  in  many  contexts),  yet  models  of  isolated  “bursty”  noise  effects  are  appropriate  in 
nearly  all  signal  processing  applications.  Second,  abrupt  changes  in  signals  occur  as  the 
result  of  system  failures  (e.g.  changes  in  dynamic  response  of  a  structure  due  to  failure  of 
some  part)  or  as  the  result  of  different  signal  propagation  paths.  Failure  detection  and 
diagnosis  for  complex  aerospace  systems  is  a  main  motivation  for  the  study  of  models  for 
abruptly  changing  signals. 


Our  work  has  made  extensive  use  of  a  particular  model  structure,  the  so-called  lat¬ 
tice  structure,  for  autoregressive  processes.  Many  advantages  of  the  lattice  structure 
have  been  recognized  for  some  time,  as  evidenced  by  the  vast  literature  that  has  been 
built  up  on  this  topic.  Nevertheless,  we  have  made  several  new  and  important  applica¬ 
tions  of  lattice  filters  in  the  research  supported  by  AFOSR  These  will  be  explained  in 
the  course  of  the  discussion  of  our  various  research  results  in  what  follows. 

A  major  portion  of  our  work  dealt  with  the  use  of  lattice  filters  in  robust  estimation 
for  AR  and  ARMA  time  series  models.  Here  a  main  accomplishment  was  to  reduce  the 
computational  complexity  of  the  iterated,  reweighted  least  squares  approach  to  outlier 
suppression;  our  first  work  on  this  was  limited  tn  AR  models  and  reported  jn  !])  Subse¬ 
quently,  we  were  able  to  expand  this  approach  to  handle  ARMA  models  as  well;  this  has 
been  reported  in  [6j,  and  a  paper  describing  the  approach  is  presently  under  revision  after 
review  |9  For  the  AR  case,  the  lattice  structure  makes  crucial  use  of  partial  autocorre¬ 
lations  to  parametrize  the  autoregressive  model,  resulting  in  a  decoupling  of  the  least 
squares  problems.  Results  in  [lj  show  that  just  one  or  two  iterations  of  the  basic  ‘‘data 
cleaning1’  procedure  provides  estimates  with  much  better  performance  than  least  squares 
The  extension  to  more  general  ARMA  models  involves  the  use  of  lattice  filters  of  length 
p+<7,  the  sum  of  the  AR  and  MA  orders,  together  with  an  important  discovery  of  the 
explicit  relationship  between  ARMA  parameters  and  partial  autocorrelations  A 

The  work  reported  in  [4]  was  directed  at  a  special  ARMA  estimation  problem, 
namely  the  use  of  the  lattice  structure  to  obtain  estimates  of  the  AR  part  of  an  ARMA 
model.  Such  a  problem  arises  when  applying  linear  prediction  (autoregressive  modeling! 
for  spectral  estimation  of  sine  waves  in  additive  white  noise  YVe  showed  in  (4;  that  it  is 
not  necessary  to  solve  a  sequence  of  least  squares  problems  (of  order  greater  than  or 
equal  to  the  autoregressive  order)  to  get  consistent  estimates;  rather  an  extended  lattice 
filter  may  be  transformed  to  give  the  estimates  from  the  solution  of  a  single  set  of  linear 
equations  (of  order  equal  to  the  moving  average  order).  This  result  is  shown  to  be  alge¬ 
braically  equivalent  to  solution  of  the  extended  Yule- Walker  equations.  However,  the 
formulation  in  terms  of  partial  autocorrelations  rather  than  in  terms  of  the  ordinary 
autocorrelation  sequence  turns  out  to  be  the  crucial  factor  in  providing  good  estimates 
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(This  was  discovered  during  the  studies  which  led  to  (9j.)  Thus  not  only  does  the  lattice 
filter  approach  provide  a  computationally  efficient  estimation  method,  hut  it  provides  a 
much  more  accurate  method  than  the  ‘‘classical’1  extended  Yule- Walker  method  Of 
course,  the  principal  advantage  still  comes  when  we  apply  the  robust  estimation  tech¬ 
niques  already  mentioned  to  this  more  general  estimation  problem,  since  the  Yule- Walker 
approach  does  not  lend  itself  to  efficient  “robustification”  by  iterated,  weighted  leas* 
squ  ares 


The  final  step  involved  in  this  research  was  the  discovery  that  some  of  the  inter¬ 
mediate  quantities  obtained  in  the  fitting  of  the  (/>+<?) — order  lattice  filter  could  be  used 
to  obtained  estimates  uf  the  MA  parameters  directly.  While  these  estimates  are  clearly 
biased,  the  fact  that  they  are  obtained  without  much  additional  computational  cost 
makes  our  approach  attractive  for  estimation  of  general  ARMA  models  in  short-to 
moderate  data  lengths  where  highly  accurate  MA  estimates  are  (essentially)  impossible  to 
obtain  by  any  means.  In  summary,  all  of  the  ingredients  have  been  put  together  for 
robust  ARMA  parameter  estimation,  as  described  in  [9f. 


Two  subsidiary  topics  were  also  the  subject  of  some  research  First,  to  visualiv 
demonstrate  how  parameter  estimation  accuracy  depends  on  AR  parameter  values,  the 
study  reported  in  »7j  was  carried  out.  In  terms  of  a  mean  square  prediction  error  distor¬ 
tion  measure,  performance  contours  were  obtained  numerically  for  second  order  models 
The  results  agree  with  intuition  obtained  gained  from  a  knowledge  of  the  Cramer-Rao 
bound  on  parameter  estimation  accuracy,  namely  that  models  with  poles  far  apart  and 
close  to  the  unit  circle  correspond  to  models  where  high  accuracy  can  be  obtained 
Second,  as  the  result  of  research  on  robust  state  estimation  for  linear  systems  (i.e.  robust 
Kalman  filtering),  a  new  linear  filtering  formulation  in  the  square-root  information  filter 
family  was  discovered.  The  method  has  computational  efficiencies  in  certain  applications, 
especially  where  singular  state  transition  matrices  are  encountered.  Work  on  the 
development  of  this  new  filter  was  recently  completed  and  will  soon  appear  [8j. 

The  final  research  topic  on  time  series  modeling  is  concerned  with  modeling  of  sig¬ 
nals  with  abruptly  changing  statistical  characteristics.  The  approach  adopted,  as 


reported  in  jlOj,  was  to  model  such  a  signal  as  piecewise- autoregressive.  Adaptive  lattice 
algorithms  for  time-varying  AR  models  have  been  widely  studied  in  the  signal  processing 
literature  The  new  feature  included  in  our  approach  to  this  problem  is  to  add  a  second 
level  of  adaptation  by  switching  the  “forgetting  factor”  in  the  adaptive  algorithm 
between  large  and  small  values  according  to  whether  or  not  a  change  in  statistics  has 
been  detected.  Various  detection  statistics  have  been  subjected  to  experimental  (and 
some  analytical)  evalu'  con,  those  chosen  are  all  readily  computed  from  signals  that 
arise  in  the  gradient  tptive  lattice  algorithm.  The  switching  between  large  and  small 
adaptation  gams  allows  close,  highly  accurate  tracking  of  parameters  during  periods  of 
stationarity  and  fast  response  to  abrupt  changes  that  are  detected 

Our  research  also  involved  a  second  general  topic,  namely  the  formulation  of  signal 
modeling  (parameter  estimation)  problems  in  a  system- theoretic  context.  We  begin  our 
description  with  some  notation  Let  (t  j.t  3,  •  )  be  a  sequence  of  observed  random 

variables  whose  probability  distributions  are  described  by  a  parametrized  family  of  den¬ 
sity  functions  •  .?/*;  #)}.  We  assume  that  there  exists  a  sequence  of  sufficient 

statistics  for  0,  (T  i(V  1),  T^V  iT  2),  -  *  ),  so  that  all  of  the  information  about  th* 
parameter  0  available  in  the  observations  up  to  instant  k  is  preserved  in  the  (fixed 
dimension!)  statistic  Tk  This  formulation  is  applicable  to  many  parameter  estimation 
problems  arising  in  applications,  including  linear  predictive  (i.e.  autoregressive)  modeling 
of  signals 

In  the  study  of  problems  where  the  observation  record  increases,  it  is  a  natural  sys¬ 
tem  theoretic  view  to  regard  a  sequence  of  sufficient  statistics  as  defining  the 
input/output  map  of  a  dynamical  system.  In  work  done  in  collaboration  with  Eduardo 
Son  tag  of  Rutgers  University,  we  have  developed  some  realization  theory  for  nonlinear 
time  varyjng  systems  that  can  be  applied  to  the  study  of  state  space  models  for  sufficient 
sequences  [2j.  The  importance  of  state  space  models  should  be  clear  they  provide  a 
recursive  means  of  updating  the  sequence  of  sufficient  statistics  as  more  observations  are 
made  Actually,  for  this  to  be  the  case,  it  is  necessary  that  the  state  space  models  be 
finite  dimensional  Results  in  [2]  indicate  classes  of  systems  for  which  finite  dimensional 
realizations  may  be  obtained  and  give  a  criterion  for  verifying  that  a  (nonlinear,  time 


varying)  state  space  model  is  a  minimum  dimension  realization  of  its  input  output  map 
The  latter  is  an  appropriately  formulated  theorem  of  the  usual  kind:  rontrollahilit y  ami 
observability  of  a  realization  imply  that  the  realization  is  minimum  dimension 

In  [2]  we  adopted  the  framework  of  nonlinear  filtering  theory',  i.e.  the  Bayesian  per¬ 
spective,  and  proved  that  3p+l  is  the  minimum  dimension  realization  of  the  recursive 
Baves  estimator  for  the  parameters  of  a  pth  order  autoregressive  process.  This  estimator 
is  based  on  reproducing  (or  conjugate)  families  of  densities,  but  it  is  equivalent,  at  least 
for  long  observation  records,  to  the  usual  least  squares  estimates  used  in  linear  predictive 
modeling 

Another  subject  considered  in  |2j  is  the  relation  of  the  notion  of  realizability  of  a 
sufficient  sequence  with  other  properties  introduced  in  the  statistics  literature,  especially 
transitivity.  We  showed  that  realizability  has  some  important  practical  implications 
through  its  connection  with  (finite  dimensional)  recursive  equations  for  updating  the 
sequence  of  statistics  that  are  not  necessarily  associated  with  transitivity 

We  followed  up  our  work  on  realizability  with  some  further  study  of  its  implies 
tions  and  limitations  The  theory  in  '2  requires  that  realizations  take  the  form 

1  =  ak(^k-l  k  1 

Tk  = 

where  the  functions  o  and  rj  are  taken  to  be  differentiable,  since  we  need  some  kind  of 
smoothness  condition  to  make  sense  out  of  the  notion  of  dimension.  In  [2j  we  gave  the 
following  example  to  show  that  there  are  sufficient  sequences  admitting  no  smooth  finite 
dimensional  realization  For  observations  from  a  stationary  Gaussian  process  with  unk¬ 
nown  mean  and  known,  nonrational  power  spectral  density  function,  there  is  a  one¬ 
dimensional  sufficient  sequence  defining  a  linear  input/output  map  that  has  no  smooth 
finite  dimensional  realization,  linear  or  nonlinear.  Viewred  from  the  perspective  of  the 
observed  process,  this  result  is  perhaps  not  unexpected,  since  there  is  no  smooth  finite 
dimensional  model  which  generates  such  observations 


In  following  up  on  this  example,  we  investigated  its  essential  feature,  namely  that 
there  is  no  finite  dimensional  model  for  the  observation  process,  and  we  showed  that  this 
is  the  underlying  feature  of  any  example  of  this  sort.  Specifically,  a  realizability  assum lo¬ 
tion,  in  conjunction  with  existence  of  a  sufficient  sequence,  implies  that  the  observation 
process  is  equivalent  to  one  that  is  generated  by  a  finite  dimensional  dynamical  system 
driven  by  a  sequence  of  independent  random  variables.  (Here  equivalence  is  in  the 
natural  sense  for  stochastic  processes,  namely  the  same  family  of  distribution  functions.) 
These  results  are  described  in  J3J,  along  with  a  discussion  of  how  parametrized  families  of 
systems  play  a  more  natural  role  for  the  notion  of  '‘finitely  generated"  stochastic  process 
than  the  approach  nnohiiig  sufficient  sequencer. 

In  further  work,  reported  in  summary  form  in  ! 5 . ,  we  have  extended  our  approach 
to  deal  with  the  problem  of  prediction  (where  the  parameters  of  the  joint  probabilit y 
density  function  of  the  observations  are  of  no  interest).  In  practice,  prediction  problem- 
usually  arise  in  connection  with  a  growing  observation  record  instead  of  a  fixed  one  As 
described  above,  if  there  exists  a  sequence  of  sufficient  statistics  for  the  parameter,  as  in 
the  case  of  an  autoregressive  process,  then  the  sequence  may  be  viewed  as  the  input- 
output  map  of  a  dynamical  system  and  the  question  of  finite  dimensional  realizability 
may  be  investigated.  For  prediction  problems,  there  are  interesting  relationships  between 
realizability  and  a  property  called  total  sufficiency.  For  autoregressive  processes,  this 
property  holds  and  provides  one  way  of  obtaining  parameter-independent  predictions. 
Unfortunately ,  it  appears  to  be  a  very  difficult  computational  task,  being  highly  ir>n- 
linear,  to  obtain  “optimum”  predictors  based  on  sufficient  statistics.  Whether  this 
approach  can  be  transformed  into  a  useful  one  by  reformulating  it  with  a  better 
parametrization,  e.g.  the  lattice  filter,  remains  as  a  challenging  problem  for  further 
research.  We  remain  convinced  of  its  importance  as  a  basic  part  of  the  understanding  of 
how  to  do  robust  prediction 
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Applications  of  Lattice  Filters 
in  Estimation  of  Time  Series  Models 

Shiping  Li 

Abstract 

This  dissertation  is  concerned  with  estimation  problems  involving  ARMA 
models  and  some  generalizations  of  AH  models  using  lattice  filter  algorithms  The 
generalizations  of  AR  models  considered  are  AH  processes  in  additive  noise  and 
piecewise  AH  processes 

After  an  introduction,  in  Chapter  2  we  present  a  new  method  to  estimate  the 
parameters  of  Gaussian  stationary  ARM  A  models  based  on  an  iterated  leas' 
squares  regression  approach  It  is  shown  that  a  (;>  +q  f-stage  lattice  whitening  filter 
can  be  used  to  obtain  consistent  estimates  of  the  AR  parameters  of  an 
ARMA^r  7)  model  It  is  also  shown  that  a  set  of  NLA  parameter  estimates  can  b« 
obtained  using  this  approach  with  little  added  computation 

In  Chapter  3.  we  consider  the  problem  of  estimating  the  AR  parameters  of  an 
ARMA  process  in  independent  additive  noise  of  two  types  For  white  Gaussian 
additive  noise,  it  is  shown  that  the  results  obtained  in  Chapter  2  can  be  used  to 
estimate  the  AR  parameters  consistently;  for  additive  outliers,  robust  estimates  of 
the  AR  parameters  are  obtained  using  a  data  cleaning  operation  to  remove  the  con¬ 
tamination 

In  Chapter  4,  we  define  a  new  model,  a  ptecewise  AR  process,  for  a  class  of 
time  senes  whose  the  statistical  properties  may  change  abruptly  (“jump  ")  at  some 
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unknown  time  points,  1  or  such  a  model  we  consider  the  problems  of  jump  detec¬ 
tion  and  fast  tracking  of  the  changing  parameter-  A  method  based  mi  the  adap¬ 
tive  least  squares  lattice  filter  algorithm  is  proposed  to  solve  the  problems  Th* 
method  automatically  detects  the  occurrences  of  jumps  and  adjusts  the  adaptation 
rate  of  the  least  squares  adaptive  lattice  algorithm,  so  both  accurate  estimates  and 
fast  tracking  ability  is  achieved 

In  Chapter  5.  we  consider  the  problem  of  joint  prediction  and  parameter  esti¬ 
mation  for  Gaussian  AH  processes  using  the  MAP  criterion  Jt  is  shown  that  the 
estimates  obtained  can  be  expressed  in  terms  of  a  set  sufficient  statistics  for  the 
parameters  and  prediction  However,  exact  MAP  estimation  is  a  difficult  nonlinear 
problem  Some  simulation  results  are  presented  to  provide  a  comparison  o: 
approximate  MAP  estimates  with  estimates  derived  from  conventional  least  square- 
methods 


